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ABSTRACT 

CHIMES is a critiquing tool that automates the 
process of checking graphical user interface (GUI) 
designs for compliance with human factors design 
guidelines and toolkit style guides. The current 
prototype identifies instances of non-compliance 
and presents problem statements, advice, and tips 
to the GUI designer. Changes requested by the 
designer are made automatically, and the revised 
GUI is re-evaluated. A case study conducted at 
NASA-Goddard showed that CHIMES has the po- 
tential for dramatically reducing the time formerly 
spent in hands-on consistency checking. Capabili- 
ties recently added to CHIMES include exception 
handling and rule building. CHIMES is intended 
for use prior to usability testing as a means, for 
example, of catching and correcting syntactic in- 
consistencies in a large user interface. 

1. INTRODUCTION 

With continuing support from the National Aero- 
nautics and Space Administration (NASA, Code 
O), the evolution of the CHIMES methodology 
and toolset has taken place in a series of research 
and prototyping cycles. The goal has always been 
to improve the usability of user interfaces devel- 
oped at the NASA-Goddard Space Flight Center 
(GSFC) by providing user-interface designers with 
an automated design-evaluation capability. Re- 
cent prototypes have focused on implementing the 
CHIMES concept of knowledge-based compliance 
checking. 
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Available user-interface design software provides 
designers with many useful capabilities, with the 
notable exception of any capability to evaluate 
the “look and feel” of a graphical user interface. 
Such interfaces are often evaluated for compliance 
with human factors guidelines or corporate style- 
guide requirements. Evaluation is typically done 
by time-consuming, manual review and usability 
testing. Taking steps to speed up the evaluation 
process, the present CHIMES prototype is capa- 
ble of evaluating the look of single and multiple 
display screens that include alphanumerics, color, 
and graphics. The full CHIMES concept encom- 
passes rule-based evaluation of user-interface be- 
havior. 

CHIMES is intended for use by GUI designers 
prior to formal usability testing, as a means of 
cleaning up a GUI and improving consistency from 
screen to screen. Rules in the knowledge base cri- 
tique the design, and an advice generator offers 
advice, warnings, and tips to the designer. Ex- 
plication of the CHIMES knowledge base and cri- 
tiquing process is the primary purpose of this pa- 
per. 

2. OVERVIEW OF CHIMES DATA FLOW 

Figure 1 provides a conceptual overview of the 
flow of data during a CHIMES evaluation. Mov- 
ing from left to right on the figure, the resource 
file representing a GUI design is acquired by 
CHIMES and transformed into an intermediate 
representation, which is transferred to the knowl- 
edge base. The acquired GUI design is then sub- 
mitted to analysis and evaluation by the user- se- 
lected rule set. Products of the analysis include 
problem statements (“critiques”), advice, and sug- 
gested modifications. User-selected modifications 
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Figure 1: CHIMES Data Flow 











are made automatically by CHIMES and sent to 
the knowledge base for re-evaluation. The re- 
source file representing the GUI design is also au- 
tomatically updated. 

The remainder of the paper focuses on the con- 
tents of the knowledge base, describes the cri- 
tiquing process, presents a case study, and dis- 
cusses plans for enhancing CHIMES. 

3. KNOWLEDGE BASE 

The knowledge base stores a representation of the 
design to be evaluated as well as the rules that 
encode the heuristics for design evaluation. Each 
rule in the CHIMES knowledge base can be con- 
sidered a critic[l]. Key components of the knowl- 
edge base include the qualitative and quantitative 
heuristics for evaluating the graphical design and 
use of color in a single display screen and, for mul- 
tiple panels, heuristics on design consistency. The 
knowledge base is implemented in CLIPS [2]. 

Graphical Display Heuristics. CHIMES 

uses guidelines from the OSF/Motif 1 Level One 
Certification Checklist[8] and from the human fac- 
tors literature to evaluate the “look” of single and 
multiple display panels. The CHIMES approach 
allows compliance checking of requirements and 
guidelines not included in the OSF/Motif defaults. 
For example, the number of type sizes and number 
of fonts per screen, as well as text justification and 
use of highlighting, can be checked for compliance 
with human factors recommendations. 

Color He uristics. The key human factors recom- 
mendation on color is that it should be used 
for functional purposes, not simply to decorate 
the screen. Functional purposes include attract- 
ing attention to critical data objects, commu- 
nicating organization, indicating status, and es- 
tablishing relationships between distant items[6]. 
To assist GUI designers in the effective use of 
color, CHIMES not only suggests appropriate 
colors but also incorporates its suggestions with 
the designer’s functional purposes for using color 
and provides remedies for misuse of color. 

Color heuristics implemented in the most recent 
prototype permit CHIMES to evaluate the consis- 
tency of color usage across multiple panels. The 
tool checks the consistency of both foreground and 

1 Motif is a trademark of the Open Software Foundation, 
Inc. 


background colors; offers alternatives to the origi- 
nal color combination; allows the designer to pre- 
view different color combinations; and permits au- 
tomatic modification of colors when the user fin- 
ishes making changes. 

The following are a few of the color heuristics ap- 
plied in a CHIMES evaluation [5]: 

• Pale foreground colors should not be 
used on a very bright saturated green 
background because of the resulting very 
low contrast. 

• The same background color should be 
used for both a panel and its items un- 
less there is a functional, user-task re- 
lated reason for using different colors. 

• Some background colors are not recom- 
mended for use with certain foreground 
colors because of the resulting color in- 
terference. 

These heuristics are implemented in dozens of 
highly specific rules. Once the detected colors have 
been evaluated, CHIMES gives specific advice to 
improve color contrast and legibility. 

Consistency- Checking Heuristics. Consistency is 
one of the primary human factors principles of 
screen design. Consistency of object location and 
screen behavior supports the end user’s develop- 
ment of expectations about where to find common 
controls and of how the GUI will respond to user 
input. In general, an interface that reliably meets 
end-user expectations supports more efficient hu- 
man performance as compared to an interface that 
violates end-user expectations. 

As a basis for checking the internal design consis- 
tency of multiple panels or screens, the CHIMES 
knowledge base contains a set of rules on which 
there is general agreement in the human factors 
literature. When departures from consistency 
are warranted in the context of user’s tasks[3], 
CHIMES is capable of handling exceptions. 

The following are a few of the consistency-checking 
heuristics implemented in the CHIMES knowledge 
base[5]: 

• The typographic elements of data items 
which serve the same type of function in 
a design are consistent within and across 
panels, unless there is a functional or 
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user-task related reason for using differ- 
ent typographic elements. 

• The background color of panels in a de- 
sign is consistent across panels, unless 
there is a functional or user-task related 
reason for using different colors. 

• The shadowing of pushbuttons is con- 
sistent within and across panels unless 
there is a functional or user-task re- 
lated reason for using different shadow- 
ing thicknesses. 

Although checking the consistency of location of 
displayed objects presents difficult technical prob- 
lems, CHIMES is capable of checking the place- 
ment of the menubar. The current criterion for 
menubar placement is that recommended by the 
OSF/Motif guidelines[8] : “at the top edge of the 
application, just below the title area of the window 
frame.” In the full CHIMES concept, other GUI 
style guides can be encoded as sets of rules in the 
knowledge base and applied upon user selection. 

4. CRITIQUING PROCESS 

The CHIMES heuristics are represented as CLIPS 
rules. A CLIPS rule has two parts: a conditional 
part and an action part. The conditional part de- 
scribes the CLIPS data-memory configuration for 
which the rule is appropriate. (The GUI design to 
be evaluated is represented as facts in the CLIPS 
data memory.) The action part of a rule specifies 
the instructions to be executed when the condi- 
tional part of the rule is satisfied. 

The CLIPS inference engine is the executor that 
determines which heuristics should be used by se- 
lecting and then executing the appropriate rule. 
Three steps are involved in selecting and executing 
rules: 1) match rules; 2) select-rules; and 3) exe- 
cute rules. In the first step, match-rules, the infer- 
ence engine finds all of the rules that are satisfied 
by the current contents of data memory according 
to the inference engine’s comparison algorithms. 
The matched rules are potential candidates for 
execution. The second step, select-rules, applies 
some selection strategy to determine which rules 
will actually be executed. The last step, execute- 
rules, fires the rules previously selected. 

Using the CLIPS inference engine and represent- 
ing the GUI design as CLIPS facts allows the rep- 
resentation of heuristics as rules to match specific 


design patterns. For example, the rule “check- 
background-color-accord-pnl” represents a way to 
check item background inconsistency. Once the 
heuristics are modeled as rules, the CLIPS infer- 
ence engine uses the rules to critique the GUI de- 
sign that has been acquired by CHIMES. 

4. CASE STUDY 

As a preliminary test of CHIMES’ ability to de- 
tect human factors problems in a user-interface 
design, we applied CHIMES to a real- world soft- 
ware application known as the Request Oriented 
Scheduling Engine (ROSE). Developed by NASA- 
Goddard, ROSE was designed to meet the needs 
of mission planners and spacecraft operators in a 
satellite ground-control environment[10]. 

The evaluation of the ROSE user interface was 
designed to meet two goals: 1) to identify human 
factors issues in need of resolution by the ROSE 
developers; and 2) to study how CHIMES can as- 
sist a GUI designer in catching and correcting hu- 
man factors problems. For comparative purposes, 
we conducted both a CHIMES evaluation and a 
heuristic (manual) evaluation [4]. 

CHIMES Evaluation of the ROSE User Interface. 
The CHIMES evaluation took less than 10 minutes 
and detected three problems related to the use of 
fonts and typographic elements. ROSE used more 
than the three fonts permitted by a conservative 
rule in the CHIMES knowledge base. Contrary 
to the convention of using normal style fonts for 
menu options, ROSE used an italic font for op- 
tions in pull-down menus. This use of italics made 
ROSE inconsistent with other OSF/Motif applica- 
tions. CHIMES ?ilso detected typographic incon- 
sistencies across widgets in ROSE. Several labels 
for the same kind of button had been implemented 
in mixed case, while others were in all upper case. 

Heuristic Evaluation of the ROSE User Interface 
Three evaluators conducted the heuristic evalu- 
ation. (Two were human factors professionals 
who specialize in user-interface design; the third 
was an experienced designer of GUIs.) They 
spent a total of 12 person hours reviewing the 
ROSE documentation and on-line demonstrations. 
The heuristic evaluation found additional prob- 
lems that CHIMES was not able to detect because 
of current limitations in its knowledge base. 

To detect some of the problems found by the eval- 
uators, CHIMES would need knowledge of user- 
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interface behavior. For example, any attempt to 
access the ROSE help facility caused the system 
to crash because this facility had not yet been 
implemented, although a help icon was displayed 
on some screens. CHIMES did not detect this 
problem because its current knowledge base en- 
compasses only the look, but not the behavior of 
buttons. The full CHIMES concept includes eval- 
uation of user-interface behavior. 

The human evaluators found problems in screen 
layout that CHIMES was not able to detect. 
In some instances, interface elements were not 
grouped to aid the user's understanding of their 
interrelationships. Further, the heuristic evalua- 
tion found that certain panel overlays obscured 
useful information. To detect problems of this 
kind, CHIMES would need semantic capabilities 
beyond its current scope. For example, CHIMES 
would need knowledge of user goals and informa- 
tion requirements in order to suggest alternative 
layouts. 

A particularly difficult issue for an automated 
evaluation is the absence of information that 
should be, but is not, displayed. For example, 
the human evaluators noted a general lack of user 
guidance (i.e., instructions displayed on the screen 
to aid the user in navigating through the ROSE 
user interface). Fairly sophisticated capabilities 
would be needed for CHIMES to detect the ab- 
sence of user guidance or other missing informa- 
tion. 

Similarly, advanced semantic capabilities would 
be needed to detect redundant information. The 
heuristic evaluation found, for example, a redun- 
dancy in panel titles, and the evaluator recom- 
mended simplifying the user interface by removing 
the redundancy. 

Problems of appropriate widget selection, iden- 
tified by a human evaluator, pose a significant 
challenge to CHIMES or any automated user- 
interface evaluation tool. For example, five pull- 
down menus were lined up horizontally to perform 
a task that should be performed by a menubar. 
Although CHIMES can detect the misplacement 
of a menubar, it cannot currently assess the ap- 
propriateness of the widgets selected by the user- 
interface designer. 

As highlighted in the case study, the capabilities 
and limitations of CHIMES make it a useful tool 


to aid the user-interface designer, but not one 
that will replace usability testing. In the realm 
of user-interface syntax, CHIMES can reliably de- 
tect both inconsistent design elements and non- 
compliance with style guidelines. With syntactic 
issues cleared up prior to usability testing, such 
testing can then concentrate on semantic issues 
that affect end-user performance and satisfaction. 

5. CURRENT AND FUTURE DIRECTIONS 

The existing CHIMES prototype reads and evalu- 
ates GUIs created in TAE Plus[9]. Although TAE 
Plus supports CHIMES development, it limits the 
designs that CHIMES can evaluate. To make 
CHIMES a useful tool to GUI designers who do 
not use TAE Plus, we are developing an interface 
to OSF/Motif’s user interface language (UIL)[7], 
which will allow CHIMES to evaluate any Motif- 
based design. 

We are also currently developing a capability to 
allow CHIMES users to customize the knowledge 
base. We have demonstrated that CHIMES can 
work with a knowledge base containing several sets 
of rules. Switching from one set of rules to another 
does not require recompiling. Further, we have 
demonstrated that a rule can be modified through 
the CHIMES user interface and that the modified 
rule can be sent back to the knowledge base for 
execution. Now we are developing a capability to 
allow CHIMES users to set up new guidelines by 
customizing existing guidelines. A new guideline 
can later be loaded into the CHIMES knowledge 
base for evaluating GUI designs. 

Other plans for the future call for research into 
possible uses for CHIMES as an intelligent agent 
and for experimental evaluation of the effects of 
CHIMES capabilities on the performance of user- 
interface designers. 
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